{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5分钟使用Python爬取天气数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "爬取网站的步骤：\n",
    "1. 设定爬取目标\n",
    "   * 目标网站：2345天气预报网 http://tianqi.2345.com\n",
    "   * 目标数据：北京2018年全年天气预报数据\n",
    "2. 分析目标网站\n",
    "   * 待爬取页面：http://tianqi.2345.com/wea_history/54511.htm\n",
    "   * 待爬取数据：数据在js里面，http://tianqi.2345.com/t/wea_history/js/201906/54511_201906.js\n",
    "3. 批量下载js文件\n",
    "   * 使用requests库实现下载，官网：https://2.python-requests.org//zh_CN/latest/user/quickstart.html\n",
    "4. 实现返回的javascript解析，得到目标数据\n",
    "   * 对于javascript的json如何解析？\n",
    "5. 将结果数据存储\n",
    "   * 将数据结果存储成csv格式，方便后续数据分析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1、构造待爬取的月份列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 构造2018全年的月份列表\n",
    "months = []\n",
    "for year in (2017,2018,2019):\n",
    "    for month in range(12):\n",
    "        months.append(\"%d%02d\"%(year, month+1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['201701',\n",
       " '201702',\n",
       " '201703',\n",
       " '201704',\n",
       " '201705',\n",
       " '201706',\n",
       " '201707',\n",
       " '201708',\n",
       " '201709',\n",
       " '201710',\n",
       " '201711',\n",
       " '201712',\n",
       " '201801',\n",
       " '201802',\n",
       " '201803',\n",
       " '201804',\n",
       " '201805',\n",
       " '201806',\n",
       " '201807',\n",
       " '201808',\n",
       " '201809',\n",
       " '201810',\n",
       " '201811',\n",
       " '201812',\n",
       " '201901',\n",
       " '201902',\n",
       " '201903',\n",
       " '201904',\n",
       " '201905',\n",
       " '201906',\n",
       " '201907',\n",
       " '201908',\n",
       " '201909',\n",
       " '201910',\n",
       " '201911',\n",
       " '201912']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "months"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2、构造待爬取的JS的URL列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "todo_urls = [\n",
    "    f\"http://tianqi.2345.com/t/wea_history/js/{month}/54511_{month}.js\"\n",
    "    for month in months\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://tianqi.2345.com/t/wea_history/js/201701/54511_201701.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201702/54511_201702.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201703/54511_201703.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201704/54511_201704.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201705/54511_201705.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201706/54511_201706.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201707/54511_201707.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201708/54511_201708.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201709/54511_201709.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201710/54511_201710.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201711/54511_201711.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201712/54511_201712.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201801/54511_201801.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201802/54511_201802.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201803/54511_201803.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201804/54511_201804.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201805/54511_201805.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201806/54511_201806.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201807/54511_201807.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201808/54511_201808.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201809/54511_201809.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201810/54511_201810.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201811/54511_201811.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201812/54511_201812.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201901/54511_201901.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201902/54511_201902.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201903/54511_201903.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201904/54511_201904.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201905/54511_201905.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201906/54511_201906.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201907/54511_201907.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201908/54511_201908.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201909/54511_201909.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201910/54511_201910.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201911/54511_201911.js',\n",
       " 'http://tianqi.2345.com/t/wea_history/js/201912/54511_201912.js']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "todo_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3、批量下载数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "datas = []\n",
    "for url in todo_urls:\n",
    "    r = requests.get(url)\n",
    "    if r.status_code!=200:\n",
    "        raise Exception()\n",
    "    # 去除javascript前后的字符串，得到一个js格式的JSON\n",
    "    data = r.text.lstrip(\"var weather_str=\").rstrip(\";\")\n",
    "    datas.append(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"{city:'北京',tqInfo:[{ymd:'2017-01-01',bWendu:'5℃',yWendu:'-3℃',tianqi:'霾~晴',fengxiang:'南风',fengli:'1-2级',aqi:'450',aqiInfo:'严重污染',aqiLevel:'6'},{ymd:'2017-01-02',bWendu:'7℃',yWendu:'-6℃',tianqi:'晴~霾',fengxiang:'南风',fengli:'1-2级',aqi:'246',aqiInfo:'重度污染',aqiLevel:'5'},{ymd:'2017-01-03',bWendu:'5℃',yWendu:'-5℃',tianqi:'霾',fengxiang:'南风',fengli:'1-2级',aqi:'320',aqiInfo:'严重污染',aqiLevel:'6'},{ymd:'2017-01-04',bWendu:'6℃',yWendu:'-5℃',tianqi:'霾',fengxiang:'北风',fengli:'1-2级',aqi:'360',aqiInfo:'严重污染',aqiLevel:'6'},{ymd:'2017-01-05',bWendu:'2℃',yWendu:'-4℃',tianqi:'霾',fengxiang:'北风',fengli:'1-2级',aqi:'280',aqiInfo:'重度污染',aqiLevel:'5'},{ymd:'2017-01-06',bWendu:'4℃',yWendu:'-2℃',tianqi:'霾',fengxiang:'南风',fengli:'1-2级',aqi:'234',aqiInfo:'重度污染',aqiLevel:'5'},{ymd:'2017-01-07',bWendu:'2℃',yWendu:'-3℃',tianqi:'小雪',fengxiang:'无持续风向',fengli:'微风',aqi:'206',aqiInfo:'重度污染',aqiLevel:'5'},{ymd:'2017-01-08',bWendu:'5℃',yWendu:'-4℃',tianqi:'阴~晴',fengxiang:'北风',fengli:'1-2级',aqi:'59',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-09',bWendu:'3℃',yWendu:'-5℃',tianqi:'晴',fengxiang:'南风',fengli:'1-2级',aqi:'42',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-10',bWendu:'4℃',yWendu:'-6℃',tianqi:'晴',fengxiang:'北风',fengli:'3-4级',aqi:'46',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-11',bWendu:'5℃',yWendu:'-5℃',tianqi:'晴',fengxiang:'南风',fengli:'1-2级',aqi:'82',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-12',bWendu:'5℃',yWendu:'-4℃',tianqi:'晴',fengxiang:'南风',fengli:'1-2级',aqi:'61',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-13',bWendu:'3℃',yWendu:'-6℃',tianqi:'晴',fengxiang:'北风',fengli:'3-4级',aqi:'26',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-14',bWendu:'3℃',yWendu:'-7℃',tianqi:'晴',fengxiang:'东南风',fengli:'1-2级',aqi:'35',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-15',bWendu:'1℃',yWendu:'-6℃',tianqi:'霾',fengxiang:'南风',fengli:'1-2级',aqi:'71',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-16',bWendu:'1℃',yWendu:'-5℃',tianqi:'霾',fengxiang:'南风',fengli:'1-2级',aqi:'123',aqiInfo:'轻度污染',aqiLevel:'3'},{ymd:'2017-01-17',bWendu:'4℃',yWendu:'-5℃',tianqi:'霾~晴',fengxiang:'北风',fengli:'1-2级',aqi:'175',aqiInfo:'中度污染',aqiLevel:'4'},{ymd:'2017-01-18',bWendu:'2℃',yWendu:'-4℃',tianqi:'多云~小雪',fengxiang:'南风',fengli:'1-2级',aqi:'59',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-19',bWendu:'1℃',yWendu:'-8℃',tianqi:'晴',fengxiang:'北风',fengli:'5-6级',aqi:'86',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-20',bWendu:'1℃',yWendu:'-8℃',tianqi:'晴~多云',fengxiang:'西南风',fengli:'1-2级',aqi:'35',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-21',bWendu:'0℃',yWendu:'-8℃',tianqi:'晴',fengxiang:'东北风',fengli:'4-5级',aqi:'55',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-22',bWendu:'0℃',yWendu:'-10℃',tianqi:'晴',fengxiang:'南风',fengli:'1-2级',aqi:'37',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-23',bWendu:'1℃',yWendu:'-8℃',tianqi:'多云~晴',fengxiang:'西南风',fengli:'1-2级',aqi:'46',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-24',bWendu:'4℃',yWendu:'-7℃',tianqi:'多云',fengxiang:'南风',fengli:'1-2级',aqi:'112',aqiInfo:'轻度污染',aqiLevel:'3'},{ymd:'2017-01-25',bWendu:'5℃',yWendu:'-4℃',tianqi:'晴~阴',fengxiang:'西南风',fengli:'1-2级',aqi:'215',aqiInfo:'重度污染',aqiLevel:'5'},{ymd:'2017-01-26',bWendu:'8℃',yWendu:'-6℃',tianqi:'多云~晴',fengxiang:'北风',fengli:'4-5级',aqi:'216',aqiInfo:'重度污染',aqiLevel:'5'},{ymd:'2017-01-27',bWendu:'5℃',yWendu:'-5℃',tianqi:'晴~多云',fengxiang:'西南风',fengli:'3-4级',aqi:'69',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-28',bWendu:'4℃',yWendu:'-4℃',tianqi:'多云',fengxiang:'南风',fengli:'1-2级',aqi:'354',aqiInfo:'严重污染',aqiLevel:'6'},{ymd:'2017-01-29',bWendu:'1℃',yWendu:'-8℃',tianqi:'多云~晴',fengxiang:'北风',fengli:'4-5级',aqi:'89',aqiInfo:'良',aqiLevel:'2'},{ymd:'2017-01-30',bWendu:'1℃',yWendu:'-5℃',tianqi:'多云',fengxiang:'南风',fengli:'1-2级',aqi:'38',aqiInfo:'优',aqiLevel:'1'},{ymd:'2017-01-31',bWendu:'5℃',yWendu:'-4℃',tianqi:'晴',fengxiang:'南风',fengli:'1-2级',aqi:'107',aqiInfo:'轻度污染',aqiLevel:'3'},{}],maxWendu:'8（2017-01-26）',minWendu:'-10（2017-01-22）',avgbWendu:'3',avgyWendu:'-5',maxAqi:'450',minAqi:'26',avgAqi:'140',maxAqiInfo:'严重污染',maxAqiDate:'01月01日',maxAqiLevel:'6',minAqiInfo:'空气优',minAqiDate:'01月13日',minAqiLevel:'1'}\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datas[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4、解析JavaScript返回的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "JSONDecodeError",
     "evalue": "Expecting property name enclosed in double quotes: line 1 column 2 (char 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mJSONDecodeError\u001b[0m                           Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-b1ec799a630b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m### 注意，这里的json是javascript格式，不能用标准库json解析\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdatas\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/json/__init__.py\u001b[0m in \u001b[0;36mloads\u001b[0;34m(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[1;32m    346\u001b[0m             \u001b[0mparse_int\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mparse_float\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    347\u001b[0m             parse_constant is None and object_pairs_hook is None and not kw):\n\u001b[0;32m--> 348\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_default_decoder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    349\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mcls\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    350\u001b[0m         \u001b[0mcls\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mJSONDecoder\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/json/decoder.py\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(self, s, _w)\u001b[0m\n\u001b[1;32m    335\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    336\u001b[0m         \"\"\"\n\u001b[0;32m--> 337\u001b[0;31m         \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_w\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    338\u001b[0m         \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_w\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    339\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/json/decoder.py\u001b[0m in \u001b[0;36mraw_decode\u001b[0;34m(self, s, idx)\u001b[0m\n\u001b[1;32m    351\u001b[0m         \"\"\"\n\u001b[1;32m    352\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 353\u001b[0;31m             \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscan_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    354\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    355\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mJSONDecodeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Expecting value\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mJSONDecodeError\u001b[0m: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"
     ]
    }
   ],
   "source": [
    "### 注意，这里的json是javascript格式，不能用标准库json解析\n",
    "import json\n",
    "json.loads(datas[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***介绍模块：demjson***   \n",
    "地址：https://pypi.org/project/demjson/2.2.4/  \n",
    "理由：  \n",
    "    It is especially useful for   \n",
    "    error checking   \n",
    "    or ***for parsing JavaScript data***   \n",
    "    which may not strictly be valid JSON data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting demjson\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/96/67/6db789e2533158963d4af689f961b644ddd9200615b8ce92d6cad695c65a/demjson-2.2.4.tar.gz (131kB)\n",
      "\u001b[K     |████████████████████████████████| 133kB 875kB/s eta 0:00:01\n",
      "\u001b[?25hBuilding wheels for collected packages: demjson\n",
      "  Building wheel for demjson (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for demjson: filename=demjson-2.2.4-cp37-none-any.whl size=73545 sha256=65de7770e49d3e3b8f38437a742798bb34bee5b9978530619cac2cc66ac27394\n",
      "  Stored in directory: /Users/peishuaishuai/Library/Caches/pip/wheels/c5/d2/ab/a54fb5ea53ac3badba098160e8452fa126a51febda80440ded\n",
      "Successfully built demjson\n",
      "Installing collected packages: demjson\n",
      "Successfully installed demjson-2.2.4\n"
     ]
    }
   ],
   "source": [
    "!pip install demjson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import demjson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'city': '北京',\n",
       " 'tqInfo': [{'ymd': '2017-01-01',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-3℃',\n",
       "   'tianqi': '霾~晴',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '450',\n",
       "   'aqiInfo': '严重污染',\n",
       "   'aqiLevel': '6'},\n",
       "  {'ymd': '2017-01-02',\n",
       "   'bWendu': '7℃',\n",
       "   'yWendu': '-6℃',\n",
       "   'tianqi': '晴~霾',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '246',\n",
       "   'aqiInfo': '重度污染',\n",
       "   'aqiLevel': '5'},\n",
       "  {'ymd': '2017-01-03',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '霾',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '320',\n",
       "   'aqiInfo': '严重污染',\n",
       "   'aqiLevel': '6'},\n",
       "  {'ymd': '2017-01-04',\n",
       "   'bWendu': '6℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '霾',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '360',\n",
       "   'aqiInfo': '严重污染',\n",
       "   'aqiLevel': '6'},\n",
       "  {'ymd': '2017-01-05',\n",
       "   'bWendu': '2℃',\n",
       "   'yWendu': '-4℃',\n",
       "   'tianqi': '霾',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '280',\n",
       "   'aqiInfo': '重度污染',\n",
       "   'aqiLevel': '5'},\n",
       "  {'ymd': '2017-01-06',\n",
       "   'bWendu': '4℃',\n",
       "   'yWendu': '-2℃',\n",
       "   'tianqi': '霾',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '234',\n",
       "   'aqiInfo': '重度污染',\n",
       "   'aqiLevel': '5'},\n",
       "  {'ymd': '2017-01-07',\n",
       "   'bWendu': '2℃',\n",
       "   'yWendu': '-3℃',\n",
       "   'tianqi': '小雪',\n",
       "   'fengxiang': '无持续风向',\n",
       "   'fengli': '微风',\n",
       "   'aqi': '206',\n",
       "   'aqiInfo': '重度污染',\n",
       "   'aqiLevel': '5'},\n",
       "  {'ymd': '2017-01-08',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-4℃',\n",
       "   'tianqi': '阴~晴',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '59',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-09',\n",
       "   'bWendu': '3℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '42',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-10',\n",
       "   'bWendu': '4℃',\n",
       "   'yWendu': '-6℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '3-4级',\n",
       "   'aqi': '46',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-11',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '82',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-12',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-4℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '61',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-13',\n",
       "   'bWendu': '3℃',\n",
       "   'yWendu': '-6℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '3-4级',\n",
       "   'aqi': '26',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-14',\n",
       "   'bWendu': '3℃',\n",
       "   'yWendu': '-7℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '东南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '35',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-15',\n",
       "   'bWendu': '1℃',\n",
       "   'yWendu': '-6℃',\n",
       "   'tianqi': '霾',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '71',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-16',\n",
       "   'bWendu': '1℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '霾',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '123',\n",
       "   'aqiInfo': '轻度污染',\n",
       "   'aqiLevel': '3'},\n",
       "  {'ymd': '2017-01-17',\n",
       "   'bWendu': '4℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '霾~晴',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '175',\n",
       "   'aqiInfo': '中度污染',\n",
       "   'aqiLevel': '4'},\n",
       "  {'ymd': '2017-01-18',\n",
       "   'bWendu': '2℃',\n",
       "   'yWendu': '-4℃',\n",
       "   'tianqi': '多云~小雪',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '59',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-19',\n",
       "   'bWendu': '1℃',\n",
       "   'yWendu': '-8℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '5-6级',\n",
       "   'aqi': '86',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-20',\n",
       "   'bWendu': '1℃',\n",
       "   'yWendu': '-8℃',\n",
       "   'tianqi': '晴~多云',\n",
       "   'fengxiang': '西南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '35',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-21',\n",
       "   'bWendu': '0℃',\n",
       "   'yWendu': '-8℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '东北风',\n",
       "   'fengli': '4-5级',\n",
       "   'aqi': '55',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-22',\n",
       "   'bWendu': '0℃',\n",
       "   'yWendu': '-10℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '37',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-23',\n",
       "   'bWendu': '1℃',\n",
       "   'yWendu': '-8℃',\n",
       "   'tianqi': '多云~晴',\n",
       "   'fengxiang': '西南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '46',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-24',\n",
       "   'bWendu': '4℃',\n",
       "   'yWendu': '-7℃',\n",
       "   'tianqi': '多云',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '112',\n",
       "   'aqiInfo': '轻度污染',\n",
       "   'aqiLevel': '3'},\n",
       "  {'ymd': '2017-01-25',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-4℃',\n",
       "   'tianqi': '晴~阴',\n",
       "   'fengxiang': '西南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '215',\n",
       "   'aqiInfo': '重度污染',\n",
       "   'aqiLevel': '5'},\n",
       "  {'ymd': '2017-01-26',\n",
       "   'bWendu': '8℃',\n",
       "   'yWendu': '-6℃',\n",
       "   'tianqi': '多云~晴',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '4-5级',\n",
       "   'aqi': '216',\n",
       "   'aqiInfo': '重度污染',\n",
       "   'aqiLevel': '5'},\n",
       "  {'ymd': '2017-01-27',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '晴~多云',\n",
       "   'fengxiang': '西南风',\n",
       "   'fengli': '3-4级',\n",
       "   'aqi': '69',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-28',\n",
       "   'bWendu': '4℃',\n",
       "   'yWendu': '-4℃',\n",
       "   'tianqi': '多云',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '354',\n",
       "   'aqiInfo': '严重污染',\n",
       "   'aqiLevel': '6'},\n",
       "  {'ymd': '2017-01-29',\n",
       "   'bWendu': '1℃',\n",
       "   'yWendu': '-8℃',\n",
       "   'tianqi': '多云~晴',\n",
       "   'fengxiang': '北风',\n",
       "   'fengli': '4-5级',\n",
       "   'aqi': '89',\n",
       "   'aqiInfo': '良',\n",
       "   'aqiLevel': '2'},\n",
       "  {'ymd': '2017-01-30',\n",
       "   'bWendu': '1℃',\n",
       "   'yWendu': '-5℃',\n",
       "   'tianqi': '多云',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '38',\n",
       "   'aqiInfo': '优',\n",
       "   'aqiLevel': '1'},\n",
       "  {'ymd': '2017-01-31',\n",
       "   'bWendu': '5℃',\n",
       "   'yWendu': '-4℃',\n",
       "   'tianqi': '晴',\n",
       "   'fengxiang': '南风',\n",
       "   'fengli': '1-2级',\n",
       "   'aqi': '107',\n",
       "   'aqiInfo': '轻度污染',\n",
       "   'aqiLevel': '3'},\n",
       "  {}],\n",
       " 'maxWendu': '8（2017-01-26）',\n",
       " 'minWendu': '-10（2017-01-22）',\n",
       " 'avgbWendu': '3',\n",
       " 'avgyWendu': '-5',\n",
       " 'maxAqi': '450',\n",
       " 'minAqi': '26',\n",
       " 'avgAqi': '140',\n",
       " 'maxAqiInfo': '严重污染',\n",
       " 'maxAqiDate': '01月01日',\n",
       " 'maxAqiLevel': '6',\n",
       " 'minAqiInfo': '空气优',\n",
       " 'minAqiDate': '01月13日',\n",
       " 'minAqiLevel': '1'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demjson.decode(datas[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tqInfos = demjson.decode(datas[0])[\"tqInfo\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'ymd': '2017-01-01',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-3℃',\n",
       "  'tianqi': '霾~晴',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '450',\n",
       "  'aqiInfo': '严重污染',\n",
       "  'aqiLevel': '6'},\n",
       " {'ymd': '2017-01-02',\n",
       "  'bWendu': '7℃',\n",
       "  'yWendu': '-6℃',\n",
       "  'tianqi': '晴~霾',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '246',\n",
       "  'aqiInfo': '重度污染',\n",
       "  'aqiLevel': '5'},\n",
       " {'ymd': '2017-01-03',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '霾',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '320',\n",
       "  'aqiInfo': '严重污染',\n",
       "  'aqiLevel': '6'},\n",
       " {'ymd': '2017-01-04',\n",
       "  'bWendu': '6℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '霾',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '360',\n",
       "  'aqiInfo': '严重污染',\n",
       "  'aqiLevel': '6'},\n",
       " {'ymd': '2017-01-05',\n",
       "  'bWendu': '2℃',\n",
       "  'yWendu': '-4℃',\n",
       "  'tianqi': '霾',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '280',\n",
       "  'aqiInfo': '重度污染',\n",
       "  'aqiLevel': '5'},\n",
       " {'ymd': '2017-01-06',\n",
       "  'bWendu': '4℃',\n",
       "  'yWendu': '-2℃',\n",
       "  'tianqi': '霾',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '234',\n",
       "  'aqiInfo': '重度污染',\n",
       "  'aqiLevel': '5'},\n",
       " {'ymd': '2017-01-07',\n",
       "  'bWendu': '2℃',\n",
       "  'yWendu': '-3℃',\n",
       "  'tianqi': '小雪',\n",
       "  'fengxiang': '无持续风向',\n",
       "  'fengli': '微风',\n",
       "  'aqi': '206',\n",
       "  'aqiInfo': '重度污染',\n",
       "  'aqiLevel': '5'},\n",
       " {'ymd': '2017-01-08',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-4℃',\n",
       "  'tianqi': '阴~晴',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '59',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-09',\n",
       "  'bWendu': '3℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '42',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-10',\n",
       "  'bWendu': '4℃',\n",
       "  'yWendu': '-6℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '3-4级',\n",
       "  'aqi': '46',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-11',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '82',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-12',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-4℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '61',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-13',\n",
       "  'bWendu': '3℃',\n",
       "  'yWendu': '-6℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '3-4级',\n",
       "  'aqi': '26',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-14',\n",
       "  'bWendu': '3℃',\n",
       "  'yWendu': '-7℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '东南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '35',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-15',\n",
       "  'bWendu': '1℃',\n",
       "  'yWendu': '-6℃',\n",
       "  'tianqi': '霾',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '71',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-16',\n",
       "  'bWendu': '1℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '霾',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '123',\n",
       "  'aqiInfo': '轻度污染',\n",
       "  'aqiLevel': '3'},\n",
       " {'ymd': '2017-01-17',\n",
       "  'bWendu': '4℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '霾~晴',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '175',\n",
       "  'aqiInfo': '中度污染',\n",
       "  'aqiLevel': '4'},\n",
       " {'ymd': '2017-01-18',\n",
       "  'bWendu': '2℃',\n",
       "  'yWendu': '-4℃',\n",
       "  'tianqi': '多云~小雪',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '59',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-19',\n",
       "  'bWendu': '1℃',\n",
       "  'yWendu': '-8℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '5-6级',\n",
       "  'aqi': '86',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-20',\n",
       "  'bWendu': '1℃',\n",
       "  'yWendu': '-8℃',\n",
       "  'tianqi': '晴~多云',\n",
       "  'fengxiang': '西南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '35',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-21',\n",
       "  'bWendu': '0℃',\n",
       "  'yWendu': '-8℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '东北风',\n",
       "  'fengli': '4-5级',\n",
       "  'aqi': '55',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-22',\n",
       "  'bWendu': '0℃',\n",
       "  'yWendu': '-10℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '37',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-23',\n",
       "  'bWendu': '1℃',\n",
       "  'yWendu': '-8℃',\n",
       "  'tianqi': '多云~晴',\n",
       "  'fengxiang': '西南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '46',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-24',\n",
       "  'bWendu': '4℃',\n",
       "  'yWendu': '-7℃',\n",
       "  'tianqi': '多云',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '112',\n",
       "  'aqiInfo': '轻度污染',\n",
       "  'aqiLevel': '3'},\n",
       " {'ymd': '2017-01-25',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-4℃',\n",
       "  'tianqi': '晴~阴',\n",
       "  'fengxiang': '西南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '215',\n",
       "  'aqiInfo': '重度污染',\n",
       "  'aqiLevel': '5'},\n",
       " {'ymd': '2017-01-26',\n",
       "  'bWendu': '8℃',\n",
       "  'yWendu': '-6℃',\n",
       "  'tianqi': '多云~晴',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '4-5级',\n",
       "  'aqi': '216',\n",
       "  'aqiInfo': '重度污染',\n",
       "  'aqiLevel': '5'},\n",
       " {'ymd': '2017-01-27',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '晴~多云',\n",
       "  'fengxiang': '西南风',\n",
       "  'fengli': '3-4级',\n",
       "  'aqi': '69',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-28',\n",
       "  'bWendu': '4℃',\n",
       "  'yWendu': '-4℃',\n",
       "  'tianqi': '多云',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '354',\n",
       "  'aqiInfo': '严重污染',\n",
       "  'aqiLevel': '6'},\n",
       " {'ymd': '2017-01-29',\n",
       "  'bWendu': '1℃',\n",
       "  'yWendu': '-8℃',\n",
       "  'tianqi': '多云~晴',\n",
       "  'fengxiang': '北风',\n",
       "  'fengli': '4-5级',\n",
       "  'aqi': '89',\n",
       "  'aqiInfo': '良',\n",
       "  'aqiLevel': '2'},\n",
       " {'ymd': '2017-01-30',\n",
       "  'bWendu': '1℃',\n",
       "  'yWendu': '-5℃',\n",
       "  'tianqi': '多云',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '38',\n",
       "  'aqiInfo': '优',\n",
       "  'aqiLevel': '1'},\n",
       " {'ymd': '2017-01-31',\n",
       "  'bWendu': '5℃',\n",
       "  'yWendu': '-4℃',\n",
       "  'tianqi': '晴',\n",
       "  'fengxiang': '南风',\n",
       "  'fengli': '1-2级',\n",
       "  'aqi': '107',\n",
       "  'aqiInfo': '轻度污染',\n",
       "  'aqiLevel': '3'},\n",
       " {}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tqInfos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 解析所有月份的数据\n",
    "all_datas = []\n",
    "\n",
    "for data in datas:\n",
    "    tqInfos = demjson.decode(data)[\"tqInfo\"]\n",
    "    all_datas.extend([x for x in tqInfos if len(x)>0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1095"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_datas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5、将结果写出到csv文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ymd': '2017-01-01',\n",
       " 'bWendu': '5℃',\n",
       " 'yWendu': '-3℃',\n",
       " 'tianqi': '霾~晴',\n",
       " 'fengxiang': '南风',\n",
       " 'fengli': '1-2级',\n",
       " 'aqi': '450',\n",
       " 'aqiInfo': '严重污染',\n",
       " 'aqiLevel': '6'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_datas[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['ymd', 'bWendu', 'yWendu', 'tianqi', 'fengxiang', 'fengli', 'aqi', 'aqiInfo', 'aqiLevel'])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_datas[0].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "with open('./beijing_tianqi_2017-2019.csv', 'w', newline='', encoding='utf-8') as csv_file:\n",
    "    writer = csv.writer(csv_file)\n",
    "    \n",
    "    columns = ['ymd', 'bWendu', 'yWendu', 'tianqi', 'fengxiang', 'fengli', 'aqi', 'aqiInfo', 'aqiLevel']\n",
    "    writer.writerow(columns)\n",
    "    \n",
    "    for data in all_datas:\n",
    "        writer.writerow([data[column] for column in columns])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
